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Abstract 

^^ 

fSJ We consider the problem of finding the number of matrices over a finite field with a certain 

rank and with support that avoids a subset of the entries. These matrices are a g-analogue of 
permutations with restricted positions (i.e., rook placements). For general sets of entries these 
numbers of matrices are not polynomials in q (Stembridge 98); however, when the set of entries 
is a Young diagram, the numbers, up to a power of g — 1, are polynomials with nonnegative 
r^ coefficients (Haglund 98). 

"t^ In this paper, we give a number of conditions under which these numbers are polynomials 

r^ in q, or even polynomials with nonnegative integer coefficients. We extend Haglund's result to 

1 " I complements of skew Young diagrams, and we apply this result to the case when the set of 

entries is the Rothe diagram of a permutation. In particular, we give a necessary and sufficient 
CO condition on the permutation for its Rothe diagram to be the complement of a skew Young 

J^ diagram up to rearrangement of rows and columns. We end by giving conjectures connecting 

^— V invertible matrices whose support avoids a Rothe diagram and Poincare polynomials of the 

QQ strong Bruhat order. 

2 1 Introduction 

^~~^ We study certain g-analogues of permutations with restricted positions, or equivalently of place- 

J> ments of non-attacking rooks. The g-analogue of permutations we work with is invertible n x n 

k^ matrices over the finite field Fq with q elements, as in |17l Ch. 1] . Then the analogue of permu- 

^ tations with restricted positions is invertible matrices over Fq with some entries required to be 

zero. 

Specifically, given a subset 5 of {1, 2, . . . , n} x {1, 2, . . . , n}, let matg(n, S, r) be the number of 
nxn matrices over Fq with rank r, none of whose nonzero entries lie in S. This is clearly an analogue 
(in the plain English meaning) of the problem of counting permutations whose permutation matrix 
has no 1 in the position of any entry of 5, but actually much more can be said. In [111 Prop. 5.1] 
it was shown that matg(n, S, r)/{q — ly is in fact an enumerative g-analogue of permutations with 
restricted positions; that is, its value, modulo {q — 1), counts the placements of r non-attacking 
rooks on the complement of S. 

The function matg(n, 5", r) can exhibit a variety of different behaviors, as seen in the following 
three examples. 

* Supported by NSF RTG grant NSF/DMS- 1148634 
■1^ Supported by a CRM-ISM Postdoctoral Fellowship 
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Figure 1: A representative matrix counted in matg(7, F, 7) where F is the complement of the point- 
hne incidence matrix of the Fano plane, shown at right. Stembridge |18] showed this to be the 
smallest example of the form matg(n, S, n) that is not a polynomial in q. 

Examples 1.1. 1. When S = 0, matg(n, 0, n) is the number of n x n invertible matrices over 
Fq, which is 

n 

i=l 

The term 13^=1 (-'- + <? + ■■■ + Q^~^) in the product is a polynomial with positive coefficients, 
and in fact is the generating series for permutations in ©„ by number of inversions. 

2. When n = 3 and S is the diagonal {(1, 1), (2, 2), (3, 3)} we have 

mat,(3, {(1, 1), (2, 2), (3, 3)}, 3) = {q - lf{q^ + 2q^ - q). 

The number of invertible nxn matrices for general n over Fg with zero diagonal was computed 
in |llt Prop. 2.2]; as in this example, it is of the form {q — !)"/((?) for a polynomial / with 
both positive and negative coefficients. 

3. When n = 7, Stembridge ^18j found a set F with 28 elements (shown in Figure [I]) such 
that matg(7, F, 7) is given by a quasi-polynomial in q, that is, by two distinct polynomials 
depending on whether q is even or odd. The set F is the complement of the incidence matrix 
of the Fano plane. 

From the examples above we see that matg(n, S,r) is not necessarily a polynomial in q, and if 
it is a polynomial in q it might or might not be of the form {q — ly f{q) where f{q) is a polynomial 
with nonnegative integer coefficients. Then a natural question to ask is the following: 

Question 1.2. What families of sets S are there such that inatq{n,S,r)/(q — ly is (i) not a poly- 
nomial in q, (a) a polynomial in q, or (Hi) a polynomial in q with nonnegative integer coefficients? 

In the remainder of this introduction, we give a summary of our progress towards answering 
this question. 



Outline and summary of results 

In Section [2j we give the definitions and notation that will be used throughout the paper including 
the definition and some properties of g-rook numbers. 

In Section^ we address general conditions on r and S under which the function matg(n, S, r) is 
always a polynomial in q. We show that if r = 1 then matq(n, S, 1) is a polynomial in q for any set 



S, though not necessarily with nonnegative coefficients. (It is an open question whether there is a 
set S such that niatg(n, S, 2) is non-polynomial in q.) Our main result of this section is to extend 
work of Stembridge [18] to give reductions to compute matg(n,S', r) in terms of smaller instances 
of similar problems when S has a row or column with very few or very many entries. 

In the rest of the paper, we discuss special families of sets S such that matg(n, S,r)/{q — ly 
is a polynomial in q with nonnegative integer coefficients. Haglund [7j showed that if the set S is 
a straight shape then m.atq{n, S,r)/{q — ly is a polynomial with nonnegative integer coefficients. 
Our second main result, proved in Section 111 is to extend this to complements of skew shapes. 



Corollary 4.6 , For any skew shape S 



'A/m; 



matg(n,5A/^,r) = {q - lyfiq), 
where f{q) is a polynomial with nonnegative integer coefficients. 

In fact, we show that this is true for an even larger class of shapes than skew shapes, namely 
those that have what we call the North-East Property. Also, because matg(n, S, r) is invariant under 
permuting rows and columns we have that matg(n, 5", r)/{q — 1)'" is a polynomial with nonnegative 
integer coefficients for any set S that is a straight or skew shape after permuting rows and columns. 

Another natural family of diagrams is the collection of Rothe diagrams of permutations, 
which appear in the study of Schubert calculus. The Rothe diagram Rw of a permutation w is a 
subset of {1, 2, . . . , n} x {1, 2, . . . , n} whose cardinality is equal to the number of inversions oiw; it 
is given by 

Rw = {{i,i) I 1 < «,i < ra, w{i) > j, w'^{j) > i}. 

See Figure ^ for some examples of Rothe diagrams. Lascoux and Schiitzenberger showed in |10j 
that the Rothe diagram i?^ of a permutation ti; is a straight shape up to permutation of rows and 
columns if and only if w, written as a word wiW2 • • • Wn, avoids the permutation pattern 2143 (i.e., 
there is no sequence i < j < k < I such that Wj < Wi < wi < Wk). Our third main result is to give 
an analogous criterion for the case of complements of skew shapes. 



Theorem 5.4\ The Rothe diagram R^ of a permutation w is, up to permuting its rows and columns, 



the complement of a skew shape if and only if w can be decomposed as w = aia2 . • . akbib2 . • . 6n-fc 
where Oj < bj for all i and j, and both 0102 . . . Ofc ^^^^ &i^2 • • • ^n-fc ^i"^ 2143-avoiding. 

We also show that this condition is equivalent to the statement that w avoids the nine patterns 
24153, 25143, 31524, 31542, 32514, 32541, 42153, 52143, and 214365, and we express the generating 
series for these permutations in terms of the generating series for 2143-avoiding permutations. 



By Corollary 4.6 , if w satisfies the condition above then matq(n, Ry^, r)/[q — ly is a polynomial 
with nonnegative integer coefficients. Surprisingly, computer calculations for n < 7 and < r < n 
[9] suggest that m.atq{n,R^,r)/{q — ly is a polynomial with nonnegative integer coefficients for all 
permutations w (see Conjecture |5.1| ). Moreover, computer calculations also suggest that when w 
avoids the permutation patterns 1324, 24153, 31524, and 426153 we have that matg(n, Rw, n)/{q — 
ly is (up to a power of q) the Poincare polynomial Pw{q) = X^u>u; g™^'"-*, where the sum is over all 



permutations u oi n above w in the strong Bruhat order (see Conjecture 6.6). Interestingly, these 
four patterns have appeared in related contexts [U El HSl [8] . 

Supplementary code for calculating matg(n, S',r) and other related objects and data generated 
by this code to test the conjectures in Section |6] are available at the following website: 



http : //sites . google . com/site/matrixf initef ields/ 
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2 Definitions 

We denote [n] = {1, 2, . . . , n}. The support of a matrix A is the set of indices {i, j) of the nonzero 
entries aij ^ 0. Fix integers n and r such that n > 1 and n > r > 0, and let S be a subset of 
[n] X [n]. We define matg (n,S',r) to be the number of n x n matrices over Fg with rank r and 
support contained in S, the complement of S. That is, matg (n, S',r) counts matrices A of rank r 
such that if {i, j) E S then aij = 0. We consider the problem of computing matq(n, S, r). 

We now define several special types of diagrams that will be important to us in what follows. 
Examples of these diagrams are given in Figure l2| We say that S* C [n] x [n] is a straight 
shape if its elements form a Young diagram of a partition. (Throughout this paper we use English 
notation and matrix coordinates for partitions.) Thus, to every integer partition A with at most n 
parts and with largest part at most n (i.e., to each sequence of integers (Ai, A2, . . . , A^) such that 
n > Ai > A2 > • • • > A„ > 0) there is an associated set S = S\ = {(i, j) | 1 < J < Aj}. We denote 
by |A| the size Ai + A2 + • • • of the shape A. This is also the number of entries in S\. Similarly, if 
A and fi are partitions such that S^ C S\ then we say that the set S\ \ S*^ is a ske'w shape and 
we denote it by Sx/fj,- Lastly, we say that a set S* C [n] x [n] has the North-East (NE) Property 
if for all i,i',j,j' G [n] such that i' < i and j < j' we have that if {i,j), {i',j), and {i,j') are in S, 
then so is {i',j')- Note that for any partitions A and fi, Sx,Sx, and Sx/f^ have the NE Property. 
But Sx/fj, in general does not have this property. 

We denote by ©„ the group of permutations on [n]. We write permutations as words w = 
W1W2 ■ ■ ■ Wn where Wi is the image of w at i. Let viiy{w) denote the number of inversions #{(?, j) | 
i < j,Wi > Wj} of w. We also identify each permutation w with its permutation matrix, the 
n X n 0-1 matrix with Is in positions {i,Wi). 

We think of the Is in a permutation matrix as n non-attacking rooks on [n] x [n] . In this case, 
the number of inversions of the permutation is exactly the number of elements in [n] x [n] that are 
not directly below/south (in the same column) or to the right /east (in the same row) of any placed 
rook. We generalize this as follows. Given a subset B of [n] x [n] (sometimes called a board) and 
a rook placement C of r non-attacking rooks on B, the SE-inversion number invsE(C', -B) is the 
number of elements in B not directly south (in the same column) or to the east (in the same row) 
of any placed rook. Then the rth (SE) g-rook number of Garsia and Remmel [Ij is 

C 

where the sum is over all rook placements C of r non-attacking rooks on B. We define the north 
east inversion number invi\iE (C,i?) and rook polynomial i?}. {B,q) analogously. 
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Figure 2: Representative matrices from matq(5, S',r) when S is (i) a straight shape, (ii) a skew 
shape, (iii) a set with the NE Property; and their respective complements (iv),(v),(vi). 

Proposition 2.1 ([!]). Given an integer partition A = (Ai, A2, . . . , A^) such that n > Ai > A2 > 
■ ■ ■ ^ -^n ^ 0, set Sx = {{i,j) I ^ ^ i ^ n,l < i < Xj}. The Garsia-Remmel q-rook number 
R^^^\Sx,q) is 



R^^^Hs, 



i=l 



n—i+1 



i + 1] 



Qi 



(2.2) 



where [ml 



l + g + 



f + ...+qm-i_ 



Remark 2.3. We will see as a corollary of Theorems 4.1 and |4.2| that for a straight shape A 



Rr {Sx,q) = R!^ {Sx,q). However, this is not true for all skew shapes. For example, if A//i 

4432/31, we have i^f ^^(54432/31,9) = I + Qq'^ + 5q^ + 3g^ + 2q^ + q^ and ^2^^^54432731,9) = 
2q + 8g^ + 7q^ + q'^. But for skew shapes in the case of n rooks we do have an analogous relation, 
as the following result shows. ■ 

Proposition 2.4. For a skew shape Sx/^x ^ [n] x [n] we have 

R^^'HSx/„q) = q("^y\^\-Rr\Sx/„g-'). 

Proof. For each rook placement of n rooks on Sx/i_i, the number of SE-inversions is equal to the 
number of inversions of the associated permutation w minus the size of /U. On the other hand, the 
number of NE-inversions of this rook placement on Sx/^ is (2) minus the number of inversions of 
w. The result follows. D 



3 General polynomiality results 

In this section, we give some general conditions under which matq(n, S, r) is a polynomial. In 
we show that for any n and S, the function matg(n, 5, 1) is polynomial in q. In 
we give reduction formulas for computing matg(n, S, r) in terms of smaller instances 



3.1 



3.2 



Subsection 
Subsection 
when S has a row or column with either very few or very many entries. 



Throughout this section we work with rectangular matrices of any dimensions rather than just 
square matrices. Thus, in this section, for integers m, n and r and a subset S of [m] x [n], we 
denote by matg(77i x n, S, r) the number of m x n matrices of rank r over Fg whose support avoids 
5. 

3.1 Polynomial formula for the rank-one case matg(ri, S", 1) 

In Figure [l] we showed an example by Stembridge [TH] of a set 5 C [7] x [7] such that matq(7, 5, 7) is 
not a polynomial in q. In this paper, we mainly focus on studying certain families of sets S where 
matg(n, S, r) is a polynomial in q. But before looking at particular sets S, we consider the rank 
r = 1 case for an arbitrary set S. 

Proposition 3.1. For any m and n and any set S C [m] x [n\, matg(r7T, x n, 5, 1) is a polynomial 
in q. 

Proof. Fix m, n and S. We count matrices with a given collection of nonzero rows. Given a 
nonempty subset T C [m] of rows, let as{T) be the number of columns with no entries which are 
both in one of the rows of T and in S. Then there are exactly (^q°-siT) _ i)(gi — l)#^~^ matrices of 
rank 1 over F^ whose support avoids S and whose nonzero rows are exactly those in T. It follows 
immediately that 

mat5(mxn,5,l)= J^ {q''^'^^'^ - l){q- l)*^'^ (3.2) 

TC[m] 
nonempty 

is a polynomial in q. D 
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(In fact one can show that if S is the diagonal {{i,i) | 1 < i < n} then matq(n x n,S,l) = 
^((2g-l)--2g" + l).) 

Remark 3.4. In later sections of this paper, we show that for certain diagrams S (straight shapes, 
skew shapes, and conjecturally Rothe diagrams of permutations) , the function matq(n, S, r)/{q — lY 
is not only a polynomial in q but also has nonnegative coefficients. However, this is not the case for 
matrices of rank 1: although each summand is a power of q — 1 times a polynomial with positive 



coefficients, the powers of g' — 1 differ. So, as in Example 3.3 negative coefficients can turn up for 
certain choices of S. Interestingly, if we substitute t = q — 1 in (3.2) we obtain a polynomial in t 
with nonnegative coefficients. Is this true more generally? ■ 

3.2 Reduction formulas when S has rows with few or many entries 

In [18^ Thm. 8.2], Stembridge gave some structural restrictions on a minimal set S such that 
matg(n, 5, n) is non-polynomial. In particular, he showed that in such a minimal example, every 
row contains at least three entries of S and at least two entries of S. In this section, we push his 
results slightly further: we show that for any rank r, if either S or S has a row with at most two 
entries then we can express matg(7ra x n, S,r) as a linear combination with polynomial coefficients 
of similar expressions for matrices of strictly smaller size. (Of course, the same arguments apply to 

6 



columns as well as to rows.) Furthermore, we explain why this approach cannot be used in the case 
where S contains three entries in some row. This does not settle the question of whether there are 
some m and n and a set S C [m] x [n] with only three entries per row such that matq(m x n, S, r) 
is non-polynomial in q, though we conjecture that such examples exist; in Stembridge's example 
[18] (see Figure [ij) the set S has four entries per row and S has three. 

For convenience, throughout this section we write expressions like matg(a x b,S,r) without 
worrying whether 5 C [a] x [b], where properly we should write matg(a x b, {[a] x [b]) Ci S, r). 

3.2.1 Reduction when S has at most t^vo entries in some ro^v 

We wish to show that if S has two or fewer entries in some row then matg(?Ti x n,S,r) can be 
reduced to a sum of polynomial multiples of similar but simpler expressions. We begin with a 
useful proposition that we use in the proof of Theorem |3.6| to do case analysis. 

Proposition 3.5. Suppose that M = [vi V2 • • • v„l is an mx n matrix over Fg of rank r and 
that the submatrix [vn-k+i ■ ■ ■ v„] has rank r' . The number of tuples w = [wi, W2, ■ ■ ■ , Wn-k) £ 
F^~^ such that 

rank 

is q'^~^ . For the other g"~^ — q"^'^ tuples, this matrix has rank r + 1. 

Proof. This is just the rank-nullity theorem: vectors w that do not increase the rank are those that 
(when augmented by k Os) lie in the row space of M. The portion of the row space with last k 
coordinates equal to is exactly the kernel of the projection onto these last k coordinates. The 
image of this projection has dimension r', so the kernel has dimension r — r', as desired. D 

Now we use this to prove the main result of this section. 

Theorem 3.6. Suppose that S C [m] x [n] contains at most two entries in the mth row. Then 
matg(m x n, 5, r) is equal to a linear combination of similar expressions for smaller matrices with 
coefficients in Z[g]. 

Corollary 3.7. // m, n and S are chosen minimal (in the sense of row- or column-removal) so 
that matg(m x n, S, r) is not polynomial in q then S contains at least three entries in each row and 
column. 
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The rest of this subsection is devoted to the proof of Theorem 3.6 



Proof. Consider the ?n,th row of the matrix-to-be: it contains zero, one or two forced Os (i.e., 
elements of S) , and either lies in a dimension-r space spanned by the first m — 1 rows or lies outside 
their (r — l)-dimensional span. We separately compute the number of matrices contributed in each 
of these three cases. 

Case 1: The set S contains no elements in the mth row. We have two possibilities: first, it might 
be that the first m — 1 rows of the matrix span a space of dimension r and the last row lies in 
this space. Then we would have matg((r?T, — 1) x n, S, r) choices for the first m — 1 rows and q^ 
choices for the last row, for a total contribution of matq{{m — l) x n, S, r)-q^ . Second, it might be 
that the first m — 1 rows span a space of dimension r — 1 and the last row lies outside this space. 
Then we would have inatq{{m — 1) x n, S, r — 1) choices for the first m — 1 rows and g" — q^~^ 



choices for the last row, for a total contribution of matq((m — 1) xn, S,r — 1) ■ (g" — q^ ^). Thus, 
the total contribution in this case is 

matg((m — 1) x n, S, r) ■ q^ + matg((m, — 1) x n, 5, r — 1) • (g" — q^~ ). 

Observe that the two instances of the function matg that appear in this expression involve 
matrices strictly smaller than m x n. 
Case 2: The set S contains one element in the ?7ith row, without loss of generality the entry (m, n). 
We have two cases depending on the dimension of the space spanned by the nth column, i.e., 
whether this column is zero or not. 

(a) We count matrices in which the nth column is the zero vector. In this case, we may choose 
the first m — 1 rows to span a space of dimension r in matg((ni — 1) x (n — 1), 5, r) ways 
and choose the last row in q^ ways, or we may choose the first m — 1 rows to span a space 
of dimension r — 1 in inatq{{m — 1) x (n — 1), 5, r — 1) ways and choose the last row in 
g"~^ — q^~^ ways. 

(b) We count matrices in which the nth column is not the zero vector. In this case, we may 
choose the first m—1 rows to span a space of dimension r in mat|j((m — 1) x n,S,r) — 



matg((7n — 1) x (n — 1), S,r) ways and (by Proposition 3.5) choose the last row in q^~ ways, 
or we may choose the first m — 1 rows to span a space of dimension r — 1 in matq((m — 1) x 
n, 5, r — 1) — matg((m — 1) x (n — 1), S,r — I) ways and choose the last row in g"~^ — g''"^ 
ways. 

The total contribution from these subcases is 

matg((m-l) x {n - l),S,r) ■ q"^ + matg{{m - 1) x {n - 1), S,r - 1) ■ {q""-^ - q'-^) + 
+ (matq((?n, — 1) x n, S,r) — matg((m — 1) x (n — 1), S,r)^ ■ q^'^ + 
+ (mat,((m-l) x n, S", r - 1) - mat5((m - 1) x (n - 1), S',r - 1)) • (g""^ - g''"^). 

Case 3: The set S contains two elements in the Tnth row, without loss of generality the elements 
{m,n — 1) and {m,n). We have three cases depending on the dimension of the space spanned 
by the (n — l)th and nth columns. 

(a) We count matrices in which the last two columns span a space of dimension 0, i.e., both 
columns are all zero. In this case, we may choose the first m—1 rows of the matrix to span 
a space of dimension r in matg((m — 1) x (n — 2) , 5, r) ways and choose the last row to lie in 
this space in q^ ways, or we may choose the first m — 1 rows to span a space of dimension 
r — 1 in matg((?n, — 1) x (n — 2), S,r — 1) ways and choose the last row in g"~^ — q^~^ ways. 

(b) We count matrices in which the last two columns span a space of dimension 1. There are 
three possible ways this can come about: the last column may be all zero and the next-to- 
last column nonzero, the next-to-last column may be all zero and the last column nonzero, 
or both columns may be nonzero but parallel. Let Si be the set that results if we remove 
the next-to-last column from S and replace each element {i, n) in S with {i,n — 1), and let 
5*2 be the set that results if we remove the last two columns from S and add a new entry 
(z, n — 1) whenever either (i, n — 1) or (i, n) appeared in S. With this notation, the matrices 
we desire to count fall into the following six classes: 

i. We may choose the first m — 1 rows so that they span a space of dimension r, the last 
column is zero and the next-to-last column is nonzero in matg((m — 1) x (n — 1), 5, r) — 
matq((m — 1) x (n — 2), S, r) ways, and by Proposition 3.5 we may extend each of these 
to an m X n matrix of rank r in q^~^ ways. 



ii. We may choose the first m — 1 rows so that they span a space of dimension r — 1, 
the last column is zero and the next-to-last column is nonzero in matg((m — 1) x (n — 
1), S,r — 1) — matg((?TT. — 1) x (n — 2), 5, r — 1) ways, and by Proposition 3.5 we may 
extend each of these to a matrix of rank r in q"~^ — g^~^ ways. 

iii. We may choose the first m — 1 rows so that they span a space of dimension r, the 
next-to-last column is zero and the last column is nonzero in matg((m, — 1) x (n — 



1), Si,r) — matg((?TT. — 1) x {n — 2),Si,r) ways, and by Proposition 3.5 we may extend 

each of these to a matrix of rank r in q^~^ ways, 
iv. We may choose the first m — 1 rows so that they span a space of dimension r — 1, 

the next-to-last column is zero and the last column is nonzero in matg((m — 1) x (n — 

1), Si,r — 1) — matq((?TT, — 1) x (n — 2), Si,r — 1) ways, and by Proposition 3.5 we may 

extend each of these to a matrix of rank r in g""^ — q''"~'^ ways. 
V. We may choose the first m — 1 rows so that they span a space of dimension r and the 

last two columns are nonzero and parallel in (q — l)(matg((?TT. — 1) x (n — 1), 5*2, r) — 

matg((m — 1) x (n — 2),S2,r)) ways, and by Proposition |3.5| we may extend each of 

these to a matrix of rank r in q^~^ ways, 
vi. We may choose the first m — 1 rows so that they span a space of dimension r and the 

last two columns are nonzero and parallel in (q — l)(matg((m, — 1) x (n — 1), 5*2, t — 



1) — matg((m — 1) x (n — 2), 5*2, r — 1)) ways, and by Proposition 3.5 we may extend 
each of these to a matrix of rank r in g"~^ — q^~'^ ways. 

(c) We count matrices in which the last two columns span a space of dimension 2. To compute 
the number of these in which the first m — 1 rows span a space of dimension r, subtract 
from matg((m — 1) x n, 5, r) the number of matrices in which the last two columns span a 
space of dimension less than 2; this number is computed in cases (a), (b)i, (b)iii and (b)v 



above. By Proposition 3.5 , we may extend each of these to a matrix of rank r in g^~^ ways. 
Alternatively, the first m — 1 rows may span a space of dimension r — 1, and the number 
of ways in which this happens is the result of subtracting the appropriate values computed 



in cases (a), (b)ii, (b)iv and (b)vi above. By Proposition 3.5, we may extend each of these 
to a matrix of rank r in g"~^ — q^~^ ways. 

As before, every instance of the function matg in each subcase is applied on matrices of size 

strictly smaller than m x n. (We omit the large, uninformative expression that is the total 

contribution from the cases above.) 

Finally, it's easy to check that these cases are exhaustive and that each yields an application of 

matg on matrices of size smaller than m x n, and that they are combined with coefficients that are 

polynomials in q, as desired. D 

3.2.2 Reductions of this sort can't work if S has three entries per row 



On first glance, it appears that the method of proof of Theorem 3.6 can be extended to show that 
the function matq(m x n, S,r) is a polynomial in q for any choice of m, n, S and r. However, as 
the example of Stembridge [18j (see Figure [I| shows, this is not the case. In this section, we briefly 
explain why this recursive approach breaks down for matrices with three or more zeroes per row. 



Suppose we attempt to recurse as in the proof of Theorem 3.6 while removing a row with three 



forced zero entries, e.g., for the set S such that 
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a64 


065 
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is a representative matrix counted in matg(6, 5, r). For instance, let's remove the bottom row. 
Then we wish to count the number of matrices of the form 



M' 



ai2 oi3 au 

022 023 

031 033 

041 a42 

051 



au 











025 











036 








046 


054 


055 






of each rank, refined by the dimension of the space spanned by the first three columns. As in Case 



3(b) of the proof of Theorem 3.6, we further refine by the geometric relationship between these 



columns; it is straightforward to check that if these columns span a space of dimension or 1, or 

if two of them are linearly dependent, then the induction goes through. The only remaining case 

is that they span a space of dimension 2 and are pairwise linearly independentjj In this case, we 

may take an appropriate linear combination of the second and third columns to eliminate the first 

column, in which case our problem appears at first glance to reduce to the problem of counting 

matrices of the form 

O13 Oi4 

023 a25 

033 ase 

a46 

054 ass 



M" 



012 

022 



042 




in which the first two columns are linearly independent; we have already shown (in Case 3(c) of 



the proof of Theorem 3.6) that under an appropriate inductive hypothesis this yields a polynomial 
answer in q. The "catch" is that the apparently valid reduction is actually wrong: in our particular 
example, the zeroes in positions (1, 1) and (2, 1) of M' , together with the elimination step, impose 
the condition that the upper- left 2x2 minor of M" is 0. This condition cannot be reduced to 
the requirement that certain entries be equal to 0, so any inductive approach of this nature would 
have to take a substantially stronger inductive hypothesis that allows us to consider restrictions 
on larger minors. While this more general question seems potentially interesting, it is quite broad 
and it seems likely that polynomiality will only hold under very restrictive conditions on the set of 
selected minors. 

3.2.3 Reduction when S has at most two entries in a row 



In this section, we show the complementary result of Theorem 3.6 and give a reduction for matq(m x 
n, S, r) when S contains at most two entries in some row. 

^The case that they are independent wiU simply be the complement of all other cases, so it is nonpolynomial in a 
minimal example if and only if our selected case is nonpolynomial. 
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Theorem 3.8. Suppose that S C [m] x [n] and that S contains at most two entries in the mth row. 
Then matq(r?T, x n, S", r) is equal to a linear combination of similar expressions for smaller matrices 
with coefficients in Z[(7]. 

Proof. As before, we split into cases depending on the number of entries of S in the mth row. 

Case 1. The set S contains no entries in the ?7ith row. In this case the mlh row is ah zero and we 
can simply remove it, so mat|j(m x n, 5, r) = matq((m — 1) x n, S, r). 

Case 2. The set S contains one entry in the Tnth row, without loss of generality the entry (m, n). In 
this case, the matrices of rank r with zeroes at the positions marked by S fall into two categories: 
those for which the (m, n) entry is also 0, of which there are matg((m, — 1) x n, S", r), and those 
for which the (m, n) entry is nonzero. In the latter case we may row-reduce, using the mth row 
to eliminate the nth column, and we find that there are {q — l)(7"matg((7Tt — 1) x (n — 1),S, r — 1) 
such matrices, where a is the number of entries of S among {(l,n), . . . , {m — l,n)}. Thus in 
total we have 

matg((m — 1) x n, 5, r) + {q — l)q"' vi\aiq{{ni — 1) x (n — 1), 5, r — 1) 

matrices in this case. 
Case 3. Suppose that the last row of S contains two entries in the mth row, without loss of generality 
the entries {m,n — 1) and {m,n). We count matrices M of rank r whose entries at positions 
marked by S are equal to 0, refining by whether the entries (m, n — 1) and (m, n) are also equal 
toO. 

(a) If both entries are zero, the number of matrices is just matq((m — 1) x n, S, r). 

(b) If the (m, n) entry is nonzero and the (m, n — 1) entry is zero the we are essentially in Case 
2 above: the entry (m, n) may be chosen in {q — 1) ways, and we may use the ?Tith row to 
eliminate the nth column, which gives a factor of q^ , where h is the number of entries of S 
among {(1, n), . . . , (?n — 1, n)}. The rest of the matrix may be filled in in matg((m — 1) x (n — 
1), S, r— 1) ways. So in total we have a contribution of ((?— l)g^ matg((m— 1) x (n— 1), 5, r— 1) 
in this case. 

(c) Similarly, if the (m, n — 1) entry is nonzero and the (m, n) entry is zero, we may apply 
the same technique to eliminate the (n — l)th column, etc. If we set c to be the number 
of entries of 5 in {(1, n — 1), ... , {m — 1, n — 1)} and let S' be the result of removing the 
(n — l)th column from S and shifting every entry in the nth column left, then we have in 
this case a contribution of {q — l)g'^matg((m — 1) x (n — 1), 5', r — 1) matrices. 

(d) Finally, if both the (m, n — 1) and (m, n) entries are nonzero (which may happen in {q — 1)^ 
ways), we use {m,n) entry to kill nonzero entries (i.e., entries of S) in the nth column. In 
this case, nonzero entries get "transferred" from the nth column to the (n — l)th; we pick 
up an extra factor of q every time there is already an entry of S in same row in the (n — l)th 
column. Let S" is the set that we get by removing the (n — l)th and nth columns from S 
and adding a new column that has an entry in row i whenever {(i, n — 1), (i, n)} C S, and 
let d be the number of i G [m — 1] such that neither {i, n — 1) nor (i, n) is in S. In this case, 
we have a total contribution of {q — l)^g'^matg((?n — 1) x (n — 1), S",r — 1) matrices. 

The total number of matrices in this case is the sum of the expressions in the four subcases. 
Each of the expressions appearing above yields a linear combination of applications of matg on 
matrices of size smaller than m x n, and the coefficients are polynomials in q, as desired. D 

This result cannot be extended to the case of three nonzero entries in some row, again by the 
example of jl8| (see Figure [I|; attempts to follow the same method of proof as in Theorem 3.8 
meet an obstruction similar to that of Section I3.2.2[ 
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Figure 3: NE elimination on a representative matrix counted in m.atq{n,B,r) with a pivot on {i,j) 
where B has the NE Property. 



Corollary 3.9. // m, n and S are chosen minimal (in the sense of row- or column-removal) so 
that matg(m x n, S, r) is not polynomial in q then S contains at least three entries in each row and 
column. 



Theorems 



3.6 



and 



3.8 



allow for the efficient recursive computation of matq(m x n,S,r) for a 
large class of sets S when m and n are small (though characterizing precisely which sets S seems 
hard). The code implemented in [U] includes these recursions. Unfortunately, for moderately large 
m and n, most sets S C [m] x [n] do not admit reductions by our theorems. 

4 Formula for matq(n, 5,r) when B has NE Property 

In [7|, Haglund proved the following result. 

Theorem 4.1 ([7j Thm. 1]). For every straight shape Sx we have 

mat,(n,5^,r) = {q - lYq\^\-'-4^^\Sx,q-'). 

We now extend this result (using the same proof technique) to all shapes with the NE Property, 
that is, with the property that for any i' < i, j < j' , if {i,j), {i',j) and (i, j') belong to B, then 
{i',j') does as well. 

Theorem 4.2. Fix any n and r and any set i? C [n] x [n] with the NE Property. The number of 
n X n matrices over Fq of rank r whose support is contained in B is 



matq{n,B,r) = {q- Ifq*'^-'' Rf^^\B,q-^). 



(4.3) 



Proof. Choose a matrix A counted in matg(n, 5, r), that is, whose support is in S, and perform 
Gaussian elimination in the following (north-east) order: traverse each column from bottom to top, 
starting with the leftmost (i.e., first) column. When you come to a nonzero entry (i.e., a pivot), 
use it to eliminate the entries to its north in the same column and to its east in the same row. See 
Figure [3] for an example of this stage of the elimination process. Then move on to the next column 
and repeat until there is at most one nonzero entry in every row and column. 

By the NE Property, at each stage of the elimination process just described we obtain another 
matrix counted in matg(n, i3,r). After elimination, the positions of the pivots are a placement of 
r non- attacking rooks on B. 

Given a fixed placement of r non-attacking rooks on B, let a be the number of cells in B that 
are directly north or directly east of a rook. There are {q — \yq°' matrices of rank r whose support 
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Figure 4: (a) Set i? with the NE Property, (b) Example of computing inatq{n,B,r) when B has 
the NE Property. There are three placements of four rooks in B with 0, 1 and 1 NE-inversions 
respectively. By Theorem 4.2, matg(4, i?,4) = {q 



l)V'-^(l + 2g-^). 



is in B that give this placement after the elimination procedure described above. It is not hard to 
see that a = ^B — r — invME(C, B). Thus, summing over all placements or r non-attacking rooks, 
we obtain 



mat,(n,S,r) = (<^ - l)^g#^- J](g-i) 



invME (C,-B) 



{q-lYq*^-^-R^:'^\B,q-^l 



C 



as desired. 



D 



Note that a priori it is not clear that the expression on the right-hand side of Equation (4.3) 



is a polynomial. However, this expression is a polynomial for the following reason: for any rook 
placement, there cannot be any more inversions than there are empty cells without rooks in them. 
There are ^B cells unoccupied by zeros, and, of these, r have rooks in them. So the maximum 
value of inv|\iE(C, B) is #i? — r. Since this is the power of q at the beginning of the formula, there 
will not be any g~^ terms, and matg(?7., -B,r) is a polynomial. 

Example 4.4. For n = 4 and r = 4, the set B = ([4] x [4]) \ {(1, 1), (3,4), (4, 1), (4,3), (4,4)} 
has the NE Property (as in Figure H[a)) and there are three placements of four rooks on B (as in 
Figure l4Fb)). The number of NE-inversions of these placements are 0, 1 and 1 respectively. Thus 

mat,(4, B, 4) = (g - lYq^^-\l + 2q-^) 
= {q-l)\q^ + 2q^). 



We give two corollaries of Theorem 4.2, First, since a straight shape S\ has the NE Property, 



by comparing Haglund's result and Theorem 4.2 we see that the (NE) and (SE) g-rook numbers of 
S\ agree. 

Corollary 4.5. For any straight shape S\ we have 

R!i'^\Sx,q)=Rf^\Sx,q). 
(Recall that in general the (NE) and (SE) g-rook numbers of a general board do not agree; see 



for example Remark 2.3 



Second, since any skew shape S\u has the NE Property, we have the following corollary: 
Corollary 4.6. For any skew shape S\jn, 



matg(n,5A/^,r) = {q- If f{q), 
where f{q) is a polynomial with nonnegative integer coefficients. 
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w = 41523 
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021 ^22 «23 «24 ^25 
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O4I ^42 O43 044 045 

051 052 053 «54 055 



If 



21534 



012 ai3 ai4 ai5 

«-21 0^22 ^23 024 025 

031 032 0-35 

041 042 043 044 045 

051 O52 O53 054 055 



w = 31524 

ai3 ai4 ai5 

021 022 O23 O24 025 

031 033 Q35 

041 042 043 044 045 

051 052 053 054 055 



Figure 5: Representative matrices counted by niatq(5, i?u;,r) where Rw is a Rothe diagram and w 
is (i) 41523 (vexillary), (ii) 21534 (skew-vexillary) , (iii) w = 31524 (not skew-vexillary) . The entries 



ttiwi are in red . 



Example 4.7. For X/fi = 4432/31, we have 

matg(4 X 4,^4432/31, S) = {q - lfq'>-\2q-^ + 8q-^ + 7q-^ + q-^) 

= {q-lfq\q + l){2q^ + 6q+l). 



In general, for skew shapes Sx/^ there is no product formula for matg(n, Sx/n,r) analogous to (2.2) 
even when r = n. 



5 Studying matg(n, 5, r) when 5* is a Rothe diagram 

Given a permutation w S &n written as a word w = -0111112 ■ ■ -Wn where Wi is the image of w at i, 
the Rothe diagram Rw is the set 



Rii 



{{hi) I 1 < «,i < ra, w{i) > j, w ^{j) > i}. 



Equivalently R^ is the set of elements in [n] x [n] that do not lie directly south or directly east of 
entries (i, Wi) of the permutation matrix of w. See Figure^ for some examples of Rothe diagrams. 
Note that i^Rw is the number of inversions of w, that is, the number of pairs {i,j) such that i < j 
but Wi > Wj. Also, Rw has the following property: if {i,j) and {k,£) are in R^ and i > k, j < i 
then the entry {k,j) is also in R^. We call this the Le property of Rothe diagramsrl 

The main conjecture for Rothe diagrams, which has been verified for n < 7 using the results in 
Section [s] [9] , is the following: 

Conjecture 5.1. // Rw is the Rothe diagram of a permutation w in &n and < r < n then 
mat q{n, Ryu, r)/(q — l)*" is a polynomial in q with nonnegative integer coefficients. 



In Subsection 5.1 we give properties of Rothe diagrams that help in calculating matg(n, R^ 



In Subsection |5.2| we study Conjecture 5.1 for the family of permutations w such that Rw is the 



complement of a skew shape (after permuting rows and columns). The conjecture holds for such 



permutations by Corollary |4.6[ In Theorem 5.4, we characterize these permutations. 



^The name "Le" was invented in [141 Sec. 6] in a context where the three entries in question formed a backwards 
letter "L" ; here we keep this terminology even though for R^ the three entries form instead the letter "F" . 
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5.1 Properties of mat q{n, S,r) when S" is a Rothe diagram 

In this section we give some simple properties of matg(n, S, r) when S = Ryj is the Rothe diagram 
of a permutation w. These properties are useful to simplify the size of computations involved in 



empirically confirming conjectures about matg(n, -R^,r) like Conjectures 5.1 and 6.6 

If the permutation w is the word W1W2 ■ ■ ■ Wn, the reverse of w is the permutation re{w) = 
WnWn-i ■ ■ -wi. The complement of w is the permutation c{w) = U1U2 ■ ■ -Un where Ui = n + 1 — 
i — Wi. In addition, the reverse complement of w is the permutation rc{w) = ^1^2 ' ' ' Vn where 
Vi = n + 1 — Wn+i-i- Lastly, the left-to-right maxima of w are the values Wi such that Wi > Wj 
for all J such that 1 < j < i. 

Proposition 5.2. Given a permutation w in ©„ and its Rothe diagram R^, we have 
(i) matg(n, i?^,r) = matq(n, i?^-i,r) and 
(ii) iaiatg{n, Rw,r) = 'niatq{n, Rrc{w),i^)- 

Proof. It is easy to see that for any permutation w, the diagram Ryj~i is the transpose of R^, and 
the first statement follows immediately. We now consider the second statement. 

Fix a permutation w with Rothe diagram R^j. Each element {i,j) of i?„, corresponds to the 
inversion of w formed by the entries with matrix coordinates {i,Wi) and {wJ ,j). In rc{w), these 
elements of w are transformed to {n + 1 — i, n + 1 — Wi) and (n + 1 — w~ ,n + l—j) and still form an 
inversion; in R^^t^\, this inversion corresponds to the element with coordinates (n + 1 — w~ ,n + 
1 — Wi). It follows immediately that the diagram Rrc(w) is the result of taking the transpose of Rw, 
rearranging rows and columns by multiplying on both sides by the permutation matrix of w, and 
rotating the result by 180°. D 

Next we characterize the indices of the rows and columns of [n] x [n] entirely contained in Rw. 
This is useful for computation because it is easy to express matg(n, 5, r) in terms of values of matg 
for sets obtained by removing rows or columns that contain no elements of S. 

Proposition 5.3. The kth column (row) of [n] x [n] is contained in R^ if and only if k is a 
left-to-right maximum of w (ofw^^). 

Proof. This follows from the definitions of Rw and of the left-to-right maxima. D 

5.2 Skew-vexillary permutations 

A permutation w is vexillary if its Rothe diagram, up to a permutation of its rows and columns, is 



the diagram of a partition. Call this partition X{w). Then by Haglund's Theorem 4.1 , for vexillary 
permutations w we have that 

mat,{n,Rw,r)=matg{n,S^^w),r) = {q - lYq''"~'''''^'"^-'Ri''^\S^),q-'). 

It is well-known that w is vexillary if and only if w avoids 2143 p!0], i.e., there is no sequence 
l<i<j<k<l<n in w such that Wj < Wi < wi < Wk. 

Next we give a characterization of permutations whose Rothe diagram, up to a permutation 
of rows and columns, is the complement of a skew shape. For such a permutation w, we have by 



Corollary 4.6 that matg(n, Ryj,r)/{q — ly is a polynomial with nonnegative integer coefficients. So 



Conjecture |5. 1| holds for these permutations. 

For the proof we need the following definition: we say that a skew shape Sxj^ in [n] x [n] is 
non-overlapping if there is no row nor column that contains entries from both S^ and S\. 
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Figure 6: If w can be decomposed as 0102 • • • afc^i&2 • • • &n- 

and b = 61^2 • • • bn-k are 2143 avoiding then Rw can be rearranged into a skew shape. 



_fc where aj < bj and both a 



aia2 ■■■ak 



Theorem 5.4. T/ie Rothe diagram of w = W1W2 ■ ■ ■ Wn is, up to permuting its rows and columns, 
the complement of a skew shape if and only if w can he decomposed as aia2 ■ ■ ■ afc&i^2 • • • bn-k where 
Ui < bj and each 0/0102 • • • Ofc 0,'^'^ ^1^2 ■ ■ ■ bn-k ^s 2143-avoiding. 

Proof. First we prove the "if" direction. This argument is ihustrated in Figure [6} Suppose that 



61 60 • • • 6- 



'n-fc 



as in the theorem statement. 



w can be decomposed into a = 0102 • • • Ofc and b 
Then the Rothe diagram R^ is block-diagonal, i.e., it consists of some entries in the upper-left 
k X k block and some in the lower-right (n — k) x (n — k) block, with no entries in the upper- 
right k X {n — k) block or lower-left {n — k) x k block. Furthermore, note that the upper-left and 
lower-right subdiagrams are identical to the Rothe diagrams of the permutations order-isomorphic 
to 0102 • • • Ofc and 6162 •• • bn-k-, respectively. 

Since both of these permutations are 2143-avoiding, and their Rothe diagrams in the upper-left 
and lower-right corners do not share any rows or columns in common, they can be rearranged inde- 
pendently to form two separate straight shapes. We may then rotate the straight shape correspond- 
ing to Rb by 180° via permuting rows and columns (without changing the rearranged upper-left 
corner) to get a straight shape in the upper-left corner and an upside-down straight shape in the 
lower-right. This is the outside of a skew shape, as desired. 

Second, we prove the "only if" direction of the theorem. Suppose that the diagram R^, when 
rearranged, forms the complement of a skew shape 5"^//^. This skew shape contains the column 
that was previously (i.e., before rearrangement) given by {{j-,wi) \ j > 1}. Likewise, it contains 
the row that was previously given by {{w^ ,j) | j > 1}. It follows that the skew shape Sx/^ is 
non-overlapping. After rearrangement, every entry of Rw either belongs to S^ or Sx. We use this 
partition of the elements of Ru, to identify the appropriate decomposition of w. 

We color an entry of Rw blue if it belongs to S"^ after rearrangement, otherwise we color it 
red. We show the following claim: for every entry Wi of u), the elements of Rw in the same row or 
column as (i, Wi) are either all blue or all red. 

Since Sx/^ is non-overlapping, the entries of R^, in each row have the same color, and likewise for 
columns. If there is an entry {i, Wi) with elements {i, j) and (fe, Wi) of R^, then by the Le property 
of Rothe diagrams {k,j) is also in R^^. Therefore all three entries have the same color, and the 
claim follows. 

By the argument of the preceding paragraph, we may color the elements of w as follows: if 
(i, Wi) is in the same row or column as a red entry of Rw then we color Wi red, whereas if {i, Wi) is 
in the same row or column as a blue entry of Rw then we color Wi blue, and otherwise we leave Wi 
uncolored. We observe a few properties of the colored and uncolored elements of the permutation. 
First, inversions of w can only happen between elements of the same color. Second, Wi is uncolored 
if and only if Wi is not involved in any inversions. And third, the subword of the blue (respectively, 
red) elements of w is 2143-avoiding. This is because by definition, the entries of Rw in the same 
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row or column as (1,11)1) for blue (respectively, red) Wi are exactly the entries in S^ (respectively, 
S\) after rearrangement. This is equivalent to saying that the subword of the blue (respectively, 
red) elements of w is vexillary and thus 2143-avoiding. 

From the three observations above it follows that the permutation w decomposes as U1C1U2C2U3 
where (i) the Ui are (possibly empty) blocks of uncolored elements, ci is the block of elements of 
one color of w, and C2 is the block of elements of the other color of w; (ii) the entries of each block 
are smaller than the entries of the following blocks, and (iii) the blocks ci and C2 are 2143-avoiding. 
Finally, if we set aia2 ■ ■ ■ a^ = uici and 6162 • • • h^-k = U2C2U3 we get a desired decomposition of w 
where Oj < bj and 0102 • • • a^ and 6162 • • • bn-k are 2143-avoiding. D 

We call the above permutations skew-vexillarjrl and we denote by X/n{w) the skew shape 
whose complement is the rearrangement of Rw£\ 



Corollary 5.5. By Theorem 4-2, if w is skew-vexillary then matg(n, i?^, r)/(g — 1)*" is equal to 



iiw{w)~rj^\ '(Sx/^{w)^Q~^)> ("• polynomial with nonnegative integer coefficients. In particular, 



Conjecture 5. 1 holds for skew-vexillary permutations. 



If tt; is a skew-vexillary permutation then every subpermutation of w is, as well. This implies 
that skew-vexillarity may be rephrased as a pattern-avoidance condition. We do this now. 

Proposition 5.6. The permutation w G 6„, can be decomposed as w = ai- ■ ■ akbi- ■ -bn-k such 
that Oi < bj for all i and j and the permutations oi • • • a^ and 6i • • • 6„_fc avoid 2143 if and only if 
w avoids the nine patterns 24153, 25143, 31524, 31542, 32514, 32541, 42153, 52143 and 214365. 

Proof. Call the decomposition in question an "SV-decomposition" (for Skew- Vexillary) . First, we 
show that if w contains any of the nine patterns listed in the statement of the theorem, it does not 
have an SV-decomposition. 

Let p be any of the eight patterns of length 5; it's easy to check that p is indecomposable, i.e., 
we cannot write p = uv with u, v nonempty and Uj < Vj for all i, j. Thus, if we write w = ab with 
Qi < bj we must have either p contained in a or p contained in b. Since p contains 2143, it follows 
that either a or 6 contains 2143, so this decomposition is not SV, as desired. 

Now consider the case of the pattern 214365. Any decomposition of w decomposes 214365, and 
it's easy to see that in any of the four decompositions of 214365, one piece or the other contains a 
copy of 2143. This completes the proof that any permutation containing the given patterns has no 
SV-decomposition. 

Now consider the converse. Suppose that w is not SV-decomposable. There are two cases. 

If w is indecomposable and contains 2143, then w contains a minimal indecomposable permuta- 
tion that contains 2143. The minimal 2143-containing indecomposable permutations are precisely 
the eight permutations of length 5 that we consider. 

Finally, we show by induction that every decomposable but not SV-decomposable permutation 
contains one of the nine patterns. Choose a such w, and write w = ab with aj < bj. Without 
loss of generality, in this decomposition we have that a contains 2143. If b has a descent, it 
follows immediately that w contains 214365. Otherwise, w = ai- ■ ■ ak{k -|- l)(fc -|- 2) • • • n. Observe 
that a permutation of this form has an SV-decomposition if and only if the shorter permutation 
a = ai ■ ■ ■ ak has an SV-decomposition; thus, a has no SV-decomposition. If a is indecomposable. 



^Note that in the Uterature [H Prop. 2.3] there is another meaning of the term "skew-vexillary permutation" 
which does not seem to be related to our definition. 

*The "function" X/fi{w) is not actually well-defined most of the time since you can switch the upper-left and 
lower-right corners by permuting rows and columns. Luckily nothing we use it for depends on this choice. 
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we have by the preceding paragraph that a contains one of the nine patterns; if a is decomposable, 
we have the same result by induction. 

Putting the two cases together, every permutation that is not SV-decomposable contains at 
least one of the nine patterns, as desired. D 

Remark 5.7. Vexillary permutations have many other interesting properties (see for example |12| 
Sections 2.6.5 and 2.8.1]). For example, the Stanley symmetric function [Ml Section 2] F^ of a 
vexillary permutation w equals the Schur function s\tyjy Since a skew- vexillary permutation w 
with skew shape X/fi{w) is SV-decomposable then F^ is the product S[n]x[n]\x^fi of Schur functions. 
Do other properties of vexillary permutations carry over to skew- vexillary permutations? ■ 

5.2.1 An enumerative aside 

Any structurally interesting class of permutations calls out to be enumerated. Vexillary permu- 
tations were enumerated by West [20] (who showed they are in bijection with 1234-avoiding per- 
mutations, which had been enumerated earlier by Gessel [6]). We now enumerate skew- vexillary 
permutations in terms of the generating function for vexillary permutations. For convenience, we 
denote by (3n(2143) the set of vexillary permutations of length n. Also, given two permutations 
u £ &i and v £ 6j, we denote hy w = u® v the permutation in &i+j defined by wt = ut for t £ [i] 
and wt+i = i + vt ioT t £ \j] . 

Theorem 5.8. Let V{x) be the ordinary generating function for 2143 -avoiding permutations and 
let SV{x) be the ordinary generating function for skew -vexillary permutations. We have 

SVix) = (1 - x)Vixf - Vix) + -^. 

I — X 

Proof. Let V{x) = l+x+2x'^+Qx'^+23x'^+. . . be the ordinary generating function for 2143-avoiding 
permutations and let I{x) = x^ + 3x^ + ISa;^ + . . . be the generating function for indecomposable 
2143-avoiding permutations of length 2 or more. Suppose w £ 6„(2143) can be written w = ui(Bu2 
where ui and U2 are nonempty permutations. If both ui and U2 contain an inversion, the four entries 
from the two inversions form an instance of 2143 in w, a contradiction. Thus, one of ui and U2 
must be the identity permutation. It follows immediately that every w £ (3n(2143) other than the 
identity can be written as w = idj ® u © id^ for identity permutations id, and id^ (possibly of length 
0) and an indecomposable 2143-avoiding permutation u. Thus, 

V{x) = h 



1-X (1-X)2 

and so 

I{x) = (l-x)V(x)+x-l. 

Now suppose that w £ &n has the property that w = u(B v where u and v are 2143-avoiding 
permutations (possibly of length 0). It follows from the preceding analysis that either w is 2143- 
avoiding itself or we can write u) = idj © ti' © idj © v' © id^ where the ids are identity permutations 
(possibly of length 0) and u' and v' are indecomposable 2143-avoiding permutations of length 2 or 
more. Thus, the generating function SV{x) for such permutations is given by 

SV{x) = V{x) + ^^ 

= {I - x)V{xf -V{x) + -^, 



as desired. D 
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-^35142 

ai3 ai4 ai5 

023 ^ 

^31 032 033 034 0^35 

041 043 044 045 

051 ^52 053 ^54 ^55 



HLi35U2) 

ai3 ai4 ai5 

a23 a24 025 

031 «32 033 «34 

041 042 043 044 

051 052 



Figure 7: Matrices indicating the (i) Rothe diagram and (ii) left hull of w 
entries aiw. are in red . 



35142. The matrix 



- inv(u) 



If w; is skew-vexillary then matg(n, Ruj,r)/{q — lY is of the form q" -mv{w)~r Y^^^^^ ^^^ 
by Corollary |5.5[ Another polynomial with this form is the Poincare polynomial of the strong Bruhat 
order (see e.g. [12^ Sec. 2.1.2]) in (3„. In the next subsection we study the connections between 
these and mat5(n, Rw^ n). 
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Poincare polynomials, matg(n, i^^y,?!] 



and g-rook numbers 



A natural question when faced with a family of polynomials with positive integer coefficients is 
whether they count some nice combinatorial object. In this section, we investigate connections 
between our polynomials matg(n, i?^, n) (note in particular that we focus on the case of full rank) 
and certain well-known polynomials we define now. 

As before, let inv(i(;) denote the number of inversions #{(i, j) \ i < j,Wi> tOj} of w. Recall the 
notion of the strong Bruhat order ^ on the symmetric group [3, Ch. 2]: if tij is the transposition 
that switches i and j, we have as our basic relations that u ~< u ■ tij in the strong Bruhat order 
when inv(ii) + 1 = inv(u • tij), and we extend by transitivity. Let Pw{q) = Z]u^«! ^™^ be the 
(upper) Poincare polynomial of w, where we sum over all permutations u that succeed w in the 
strong Bruhat order. Equivalently, Pw{<l) is the rank generating function of the interval [i(;,i(;o] in 
the strong Bruhat order where vjq is the largest element nn — 1 ••• 21 of this order. 

Example 6.1. If tD = 3412, then the permutations in 64 that succeed w in the Bruhat order 
are 3412,3421,4312 and 4321. The generating polynomial for this set by number of inversions is 

^2143(q)=9^ + 2q5+(7'. 

In [15], Sjostrand gave necessary and sufficient conditions for Pw{q) to be equal to a g-rook 
number of a skew shape associated to w. Namely, the left hull Hl[w) of w is the smallest 
skew shape that covers w. Equivalently, Hl(w) is the union over non-inversions {i,j) of w of the 
rectangles {{k,£) \ Wi < k < Wj,i < £ < j}. See Figure [tI for an example of the left hull of a 
permutation. 

The following special case of a result by Sjostrand characterizes when Pwig) is equal to the rook 
polynomial of the left hull of the permutation w. 

Theorem 6.2 ( |15t Cor. 3.3]). The Bruhat interval [w,Wo] in (3„ equals the set of rook placements 
in the left hull Hl{w) of w (and in particular Rn {HL{w),q) = q'^^'Rwiq) where /i is the shape 
such that Hl{w) = S\i^^ for some X) if and only if w avoids the patterns 1324,24153,31524, and 
426153. 



If tt; is a skew-vexillary then by Corollary 5.5 we know that matq(n, R^, n)/{q — 1)" is (up to a 
power of q) a g-rook number. Next we show that this g-rook number is essentially a g-rook number 
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Figure 8: Example of Proposition |6.3[ For the permutations w and v shown we have that 
matg{5,R^,5)/{q - l)^ = ^^^--M . p^(g). 



of the left hull of a permutation v that avoids the four patterns above. Therefore by Theorem 6.2 
mat(n, Ry^, n)/{q — 1)*^ is (up to a power of q) a Poincare polynomial Pv{q)- 

6.1 matq (n, i?^,n) for skew-vexillary permutations is a Poincare polynomial 



In this section we use Sjostrand's result (Theorem 6.2 ) to show that for skew-vexillary permutations 
w, the function matg(n, Rw,n)/{q — 1)" is not only a polynomial with nonnegative coefficients but, 
up to a power of q, is a Poincare polynomial. 

Proposition 6.3. // w is skew-vexillary then 

mat,(n, i?^,n) = qi^h'^''^-') (q - 1)" • p,{q) 
for some permutation v G <Sn- 



Proof. If w is skew-vexillary, then by Corollary 5.5 we know that 



matq{n,Rw,n)/{q- 1)^ 



-mv(«))-7io{NE)/c' , ^ „-1n 
^n l'-'A//t(«))i y A 



where -R^'^^^(5'a/^(^),(?) is the rook polynomial of Sx/^(^yj^, the non-overlapping skew shape whose 
complement is the rearrangement of Rw We will show that this polynomial is the Poincare poly- 
nomial Pviq) of a permutation v. 

Define the permutation matrix of v as follows: let w = aia2 ■ ■ ■ afc6i&2 ■ ■ ■ &n-fc be the decompo- 
sition promised by Theorem 5.4 Let A = (Ai, . . . , A^) and yu = (//i, . . .). For i = 1, . . . ,k, let Vi = 
mm{{[n]\[iJ,i])\{vi, . . . ,Vi^i}) and for j = 1, . . . ,n- A; let Un+i-j = max([A„_j] \{i;„_j+i, . . . , v„}). 
This defines a 0-1 matrix with exactly one 1 in every row; it follows from the proof of Theorem |5.4| 
that this matrix is in fact a permutation matrix (with {vi, . . . ,Vk} = [k] and {vk+i, ■ ■ ■ ,Vn} = 
{n—k+1, . . . , n}). See Figurelslfor an example of this construction. It is clear that Sx/^{w) = Hl{v). 



Also by Proposition 



2.4 



?(NE) 



By construction the prefix vi 



we have that q^'^)~^^^Rir^' {Sx/^,iw),q-') = R^ 



■ Vn avoids 213 and Vj > Vi 



■ Vk avoids 132 and the suffix Vk+i 
for j > k + 1 and i < k. It is easy to see that the set of permutations with such a decomposition is 
closed under containment of patterns, and does not contain any of the permutations 1324, 24153, 
31524, and 426153. Therefore, every permutation in this set, and in particular v, avoids these four 
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patterns. By Theorem 6.2 it follows that q'^'Rn {HL{v),q) = Pv{q)- Thus 



matg(n,i?^,n)/(g-l)" = g"'-'-("')-"i?i^E)(5,/^(^),<7-i) 

= g(2)-i-(-)(g-l)".p,(g), 

as desired. D 



Example 6.4. By Theorem 5.4 the permutation w = 21534 is skew-vexillary. After rearranging 
rows and columns (see Figure [SJ), the skew shape Sx/^{w) is S^^^^^/i. The associated v is 21453 and 
we have 

matq(5, i?2i534, 5) = q^^-^{q - lfP2U53{q) 

= q'^iQ - l)^(g^° + V + 9g^ + iV + ISg*' + llg^ + 5q^ + q^). 

Remark 6.5. Note that the result above does not hold for all Rothe diagrams. There exist 
permutations w for which there does not exist any permutation v such that in.atq{n,Ryj,r) = 
q{2)-'M^){q-lYP^(q). For example, take w = 31524 (see Figure ^(iii) and Table[l|). In this case 

mat5(5, ii3l524, 5) = q^{q - lf{q^^ + 4q^ + 9q^ + 12g^ + Wq^ + 5q^ + q^). 

One can show (either by computer search or by a direct argument about the possible structure of 
the inversions) that there is no permutation v in S5 such that Pv{q) = q^^ + 4g^ + 9q^ + 12g^ + 
lOq^ + 5q^ +q'^. ■ 

We have shown that for a skew-vexillary permutations w, matq(n, i?^,n)/(g — 1)" is equal (up 
to a power of g) to the Poincare polynomial of some permutation v. Next we consider the problem 
of classifying permutations w such that matg(n, R^, n)/{q — 1)" is equal (up to a power of q) to the 
Poincare polynomial of the same permutation. 

6.2 Further relationships between m.a.tg{n,Rw,n) and Poincare polynomials 

Computational evidence for n < 7 [9j suggests the following conjecture. 



Conjecture 6.6. Fix a permutation w in Gn o,nd let Rw he its Rothe diagram. We have that 
Taa.tg{n,Ru,,n)/{q — 1)" is coefficient-wise less than or equal to gV2J~™^('")p^(g). We have equality 
if and only if w avoids the patterns 1324, 24153, 31524, and 426153. 



Remark 6.7. The patterns that appear in Conjecture 6.6 and in Theorem 6.2 are the same. Also, 
the reverses 4231,35142,42513, and 351624 of these patterns have appeared in related contexts in 
a conjecture of Postnikov [14] proved by Hultman-Linusson-Shareshian-Sjostrand [8], and in work 
by Gasharov- Reiner [3]. This suggests further interesting connections. These permutations were 
recently enumerated by Albert-Brignall [1]. ■ 

The values of the three polynomials matg(n, Rw,n)/{q — 1)", Pw{q), and Rn {HL{w),q) when 
w is equal to the four patterns of Conjecture |6.6| are shown in Table [TJ In these cases the three 



polynomials are all different. By Theorem 6.2 and Theorem |4.2[ Conjecture 6.6 is equivalent to the 
following: 
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w 


1324 


24153 or 31524 


matg (n,i?^,n) 


q^ + Sq"^ + 5q^ + 5q^ + 3q^ + q 


^10 j^ 4^9 + 9^8 ^_ i2q^ ^ j^OgS + 5g5 _^ ^4 


(?-l)"9fe 


P^{q) 


q^ + 3g^ + 5g4 + 6q^ + Aq^ + q 


glO j^ 4^9 _^ 9^8 _^ i^q7 j^ j^^^e _^ 5^5 j^ q4 


q-R^^^\HLiw)) 


q^ + 3q^ + 5q* + 6q^ + 5q^ + 3q + 1 


qW + 4^9 ^ 9^8 ^ ;L3g7 ^ ^2q^ ^ 7^5 ^ 2q4 



w 



426153 



matg (n,i?^^,n) 


gi5 + 5gi4 + 14^13 + 24qi2 + 27g" + l%^° + Tg^ 


(q-l)"?'= 


i'™(g) 


gi5 + 5gi4 + 14^13 _^ 25<?i2 + 28g" + 19gi'' + 7q^ 


q-R'^^^\HL{w)) 


gi5 + 5gi4 + 14^13 _^ 25qi2 + 29g" + 21gio + Sg^ 



Table 1: For the four special patterns w of Conjecture 6.6 we give matg(n, i2^,n)/((g — l)"q'^) 

where k = Q) — inv(i/;), the Poincare polynomials Pw{q), and (/"-Rn \HL{w),q) where a is the size 
of the subtracted partition of the skew shape Hl{w). 



Conjecture 6.8. Fix a permutation w in (3„, let R^ he its Rothe diagram and let Uw = n'^ — 
#Hl{w) — mv{w). We have that uiatq{n,Rw,n)/{q — 1)" is coefficient-wise less than or equal to 



q"-"^ matg(n, HL{w),n)/{c 
31524, and 426153. 



We have equality if and only if w avoids the patterns 1324, 24153, 



This conjecture is not true for matrices of lower rank. For example, for w 
matg(2, i?2i, l)/{q - 1) = 2g + 1 and matg(2, Hl{21), l)/(g - 1) = 2. 



21 G ©2 we have 



Remark 6.9. If Conjecture 6.8 holds then by Theorem 4.2 and |lll Prop. 5.1] it follows that 
whenever w avoids the four patterns, the shapes R^ and Hl{w) have the same number of placements 
of n non-attacking rooks. This is not obvious since for such permutations the shapes are distinct 
even after permuting rows and columns. Moreover, computer experiments for n < 7 [9] suggest 
that the converse is also true, i.e., if w contains any of the four patterns, the shapes have different 
numbers of rook placements. This apparent equivalence of necessary and sufficient conditions 
between the "g case" and the "g = 1 case" does not necessarily hold in similar settings (see p^ 
Thm. 7] and [H Thm. 3.4] for an example). ■ 

We end by giving a very preliminary step in proving these conjectures. 

Proposition 6.10. Ifw is a 12>24:- avoiding permutation then the complement of Hl{w) has at least 
as many entries as the Rothe diagram R^ of w. 

Proof. Let lu be a 1324-avoiding permutation. We give a one-to-one map ip between the entries of 
the Rothe diagram R^j and the complement of the left hull Hl{w). 

Given an entry {i,j) in Ryj we have two possibilities: either there is or there is not an entry 
{k,Wk) of vu such that k < i and Wk < j (i.e., an entry of w NW of {i,j))- Let A^ be the set of 
entries of R^ of the first type and let S^j be the set of entries of the second type. If {i,j) S A^ 
then define ^{i,j) = {i,j)- If instead {i,j) G B^ then define (p{i,j) = {w~ ,Wi). See Figure 9 for 
an illustration of (p. We show that (p is well-defined and injective. 

Choose {i,j) in Ry^. There are no entries of w above {i,j) in the same column or to its left in 
the same row. If in addition {i,j) is in A^j then by definition of the left hull the entry {i,j) is not 
in Hl{w). In this case ip{i,j) = (i,j) G Hi{w) as desired. 
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Figure 9: Example of Proposition 6.10 For the permutation w = 4132, the Rothe diagram is Rw = 
{(l,l),(l,2),(l,3),(3,2)}andtheleft-huUFZH = {(l,l),(l,2),(l,3),(2,4),(3,4),(4,3),(4,4)}. 
The map ip is given by (1, j) i— )• (l,i) for j = 1, 2, 3 and (3, 2) i— )• (4, 3). 



On the other hand, if {i,j) is in B^j then there is some entry (k, Wk) of w with k < i and Wk < j- 
Since w is 1324- avoiding, there can be no entry [i, we) of w such that i > wj and wg > wi. Thus, 
^{i,j) = {w~ ,Wi) G Hl{w). This completes the proof that the map (p is well-defined. 

Finally, we show that 99 is one-to-one. Since ip is defined piecewise it is enough to show that (p is 
one-to-one on A^] and B^ and that <pi{Ayj) and ip{Bw) are disjoint. The injectivity on Ayj is trivial. 
The injectivity on Byj follows since w is a permutation and so {w~ ,Wi) uniquely defines (i,j). 

Moreover, Hl{w) has two components; (p{Aw) is the NW component while ^{B^) is contained in 
the SE component, so the images are disjoint. This completes the proof that ^ is one-to-one. D 
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